This application is based on applications Nos. 10-012532 and 10-029691 filed in Japan, the contents of which is hereby incorporated by reference.
1. Field of the Invention
The present invention relates to pixel interpolating devices and more particularly to a pixel interpolating device using the IM-GPDCT method for interpolating a pixel while restoring a high frequency component of an image.
2. Description of the Related Art
In converting a pixel density (interpolating a pixel) of an image based on image information included in a sampled original image, a method (xe2x80x9cIM-GPDCT methodxe2x80x9d) is conventionally known which restores a spatial high frequency component, which is lost during a sampling process, under the two restrictive conditions that information on a passing frequency band is correct and an expanse of an image is limited in a process in which normal transformation and inverse transformation of the image are repeated by orthogonal transformation.
The principle of the method will be described in the following. An operation is known which restores an original signal that is lost because a frequency band is limited when an original image is sampled. Such an operation is generally accompanied by the super-resolution problem.
In any observation system that can physically be implemented, a high frequency component of at least a certain frequency cannot be observed.
For example, an image pick-up system has a limited size of an entrance aperture, and the image pick-up system itself functions as a low pass filter (LPF). Thus, a large number of frequency components that can be propagated are lost, and resolution is lowered.
The lost resolution can be obtained only by bandwidth extrapolation (super-resolution problem) in which an original signal prior to passage through the image pick-up system is found from an image signal that can be obtained through the image pick-up system.
The super-resolution problem is mathematically formulated for a function of one variable as described below. When an original signal in a real space region is f(x), a signal that is formed by limiting the frequency component band of original signal f(x) to cut-off frequency u0 at most and that actually passes through an image pick-up system is g(x), and the process for carrying out band limitation is expressed as A, the expression (1) below is derived.
g(x)=Af(x)xe2x80x83xe2x80x83(1)
The process A corresponds to actual application of an LPF by passing the original signal through the image pick-up system.
The functions that correspond to Fourier transform of signals f(x) and g(x) above are assumed to be F(u) and G(u), and a window function W(u) in a frequency region is defined by the following expressions (2) and (3).
W(u)=1(|u|xe2x89xa6u0)xe2x80x83xe2x80x83(2)
W(u)=0(|u| greater than u0)xe2x80x83xe2x80x83(3)
Performance of window function W(u) corresponds to application of an ideal LPF.
Further, the expression (1) above is expressed in a frequency region as the expression (4) below.
G(u)=W(u)F(u)xe2x80x83xe2x80x83(4)
The super-resolution problem is intended to find original signal f(x) from band-limited signal g(x) by the expression (1) in a real space region and to find F(u) from G(u) of the expression (4) in a frequency region.
If original signal f(x) is not limited at all, however, F(u) cannot be found.
Accordingly, the super-resolution problem can be solved by applying a process in which unlimited resolution can be obtained in principle when original signal f(x) is subjected to spatial region limitation so that an object has a limited size, and f(x) only exists in a certain region, a region between xe2x88x92x0 and +x0, for example, and it does not exist outside the region.
Conventionally, the Gerchberg-Papoulis iteration method (GP iteration method) is used to solve the super-resolution problem.
FIG. 14 illustrates the principle of the GP iteration method. In FIG. 14(A), (C), (E) and (G) correspond to a frequency region while (B), (D), (F) and (H) correspond to a real space region. FIG. 14(B) shows original signal f(x) of which region is limited to a space |x|xe2x89xa6x0. FIG. 14(A) shows Fourier transform F(u) of original signal f(x), and F(u) includes even an unlimitedly high frequency component because the region of original signal f(x) is limited.
FIG. 14(C) indicates that only G(u), which is the part of the space |u|xe2x89xa6u0 of F(u), is observed. In other words, the expression (4) using a window function such as the expressions (2) and (3) above is formed.
Inverse Fourier transform of G(u) is g(x) in FIG. 14(D). Solving the super-resolution problem is to find F(u) or f(x) from G(u) above.
The operation in the GP iteration method will be described in the following. Since the band of G(u) is limited to |u|xe2x89xa6u0, g(x) extends unlimitedly.
Since it is known that the region of original signal f(x) is limited to the interval |x|xe2x89xa6x0, however, the same region limitation is performed even on g(x).
In short, only the part of interval |x|xe2x89xa6x0 in g(x) is extracted to obtain f1(x). When f1(x) is expressed as an expression that uses a window function w(x) expressed by the following expressions (5) and (6), the expression (7) is obtained. This is function f1(x) shown in FIG. 14(F).
w(x)=1(|x|xe2x89xa6x0)xe2x80x83xe2x80x83(5)
w(x)=0(|x| greater than x0)xe2x80x83xe2x80x83(6)
f1(x)=w(x)g(x)xe2x80x83xe2x80x83(7)
Fourier transform of f1(x) results in F1(u) in FIG. 14(E). Since the region of f1(x) is limited, F1(u) extends unlimitedly. However, a correct value of G(u)=F(u) is already known for space |u|xe2x89xa6u0, and therefore the portion of |u|=xe2x89xa6u0 in F1(u) is substituted by G(u).
The waveform formed in this manner is G1(u) in FIG. 14(G). The relations are expressed by the expressions (8) to (10). Inverse Fourier transform of G1(u) above is g1(x) in FIG. 11(H).
G1(u)=G(u)+(1xe2x88x92W(u))F1(u)xe2x80x83xe2x80x83(8)
G1(u)=G(u)(|u|xe2x89xa6u0)xe2x80x83xe2x80x83(9)
G1(u)=F1(u)(|u| greater than u0)xe2x80x83xe2x80x83(10)
The processing from (C), (D) to (G), (H) in FIG. 14 is the first round of the GP iteration method. Then, the operation of extracting only the portion of interval |x|=xe2x89xa6u0 from g1(x) in FIG. 14(H), carrying out Fourier transform on f2(x) (not shown) corresponding to f1(x) in FIG. 14(F), and finding F2(u) (not shown) corresponding to FIG. 14(E) is repeatedly performed. Thus, an original signal can perfectly be restored.
Conventionally, an operation load is reduced by substituting Fourier transform in the GP iteration method by discrete cosine transform (DCT). This is called the xe2x80x9cIM-GPDCTxe2x80x9d method.
FIG. 15 is a flow chart schematically showing a processing flow carried out in image magnification processing (an example of pixel interpolation processing) using the conventional IM-GPDCT method, and FIG. 16 illustrates the processing of the flow chart in FIG. 15.
It is assumed here that an original image consisting of Nxc3x97N pixels shown in FIG. 16(A) is magnified m times to produce an image of (Nxc3x97m)2 pixels. In FIG. 16, the numbers in parenthesis correspond to the step numbers of the flow chart in FIG. 15.
Referring to FIG. 15, the number of iteration times in the GP iteration method and the value of a magnification rate (resolution conversion rate) are set in step S1. In step S2, an original image to be magnified, shown in FIG. 16(A), is read. In step S3, an image of interest (herein, an image shown in FIG. 16(A)) is extracted.
In step S4, an image extending around the image of interest of Nxc3x97N pixels (extension region) is found to limit the spatial expanse of the image. Conventionally, the data of an image to be extended is fixed to a particular value, and calculation of extension region data is not carried out. That is, in step S4, predetermined image data xe2x80x9cLxe2x80x9d is added as an extension region to the original image, and expansion to an image of nNxc3x97nN pixels shown in FIG. 16(B) is performed. Here, n is a real number larger than 1, and n is set so that nmN is a power of 2.
In step S5, the image in FIG. 16(B) is transformed to a frequency component a shown in FIG. 16(C) by two-dimensional DCT transform. The frequency component a is known information in the DCT region and corresponds to a spatial low frequency component.
In step S6, the value of a is stored. In step S7, the frequency band of frequency component a is extended to a high frequency band according to a magnification rate, as shown in FIG. 16(D).
At this time, the high frequency band for expansion is set to an initial value 0. The extended frequency region is set to have nmNxc3x97nmN pixels.
In step S8, inverse DCT (IDCT) is carried out on the frequency region extended as shown in FIG. 16(D) to be transformed to an image region. At this time, the image region has an image size of nmNxc3x97nmN, and a portion a of mNxc3x97mN pixels at the center is a magnified image.
In step S9, the number of iteration times is updated. In step S10, the region, indicated by xc3x97 signs, outside the mNxc3x97mN-pixel portion xcex1 at the center in FIG. 16(E) is corrected to a not-clear but predetermined value xe2x80x9cLxe2x80x9d by IDCT. Thus, the state of FIG. 16(F) is attained.
This operation is called spatial region limitation. When DCT is carried out on the image in FIG. 16(F) having the corrected extension region in step S11, the frequency component b shown in FIG. 16(G) can be obtained.
In step S12, a low frequency region of the frequency component b obtained in step S11 is substituted by a known value a to attain the state of FIG. 16(H).
In step S13, IDCT is carried out on the region including frequency components a and b to obtain the image in FIG. 16(I). In step S14, a determination is made as to whether the number of iteration times exceeds a preset value and, when it does not, the processing from step S9 to step S13 is repeatedly performed.
When the number of iteration times exceeds the value in step S14, the magnified image is output in step S15, and all the operation is completed.
In the conventional technique described above, an original image is not divided but it is transformed at a time. When DCT transform is carried out on a large sized image, however, enormous processing time is required, which makes the conventional method non-practical.
Accordingly, the method of once dividing an original image into small sized image blocks and then carrying out resolution conversion processing in each block has been proposed.
FIG. 17 schematically shows how an original image block is cut out and an extension region is set in the conventional IM-GPDCT processing.
Referring to FIG. 17, an original image (#601) is divided into images of predetermined Nxc3x97N pixels (#602) by block division processing. Here, the cut-out block to be processed is called a target block (#603). The entire original image is processed by causing all blocks to be target blocks. A case where a block near the center of the character  in the original image is cut out will be described as an example.
An extension region of nNxc3x97nN pixels is added to the target block (#604), and the resolution conversion processing thereafter is carried out (#605).
The extension region data is fixed to a particular value as described above. In the conventional method, xe2x80x9c0,xe2x80x9d xe2x80x9c255,xe2x80x9d or the average value of image data in the target block is generally set as the extension region data.
FIG. 18 shows three-dimensional image data in a target block. Here, a reflection factor is adopted as image data, and the target block is formed of eight pixels in both of main and sub scanning directions.
As is apparent from FIG. 18, the reflection factor is higher on the far left side and lower toward the near right side in the image data in the target block.
In the following, problems with the conventional technique will be described based on a case where an image on the cross section (cross section A indicated by the dashed line in the figure) of the fourth pixel in the sub scanning direction of the target block is to be processed.
FIG. 19 shows charts for describing problems with a case where image data (reflection factor) in the extension region is set to 255 and the IM-GPDCT processing is carried out in the conventional technique.
Referring to the figure, a) shows relations between a pixel position and its image data on cross section A of the target block in FIG. 18, and b) shows a state where image data L=255 is added to the extension region of the original image. In this case, the extension region data has an unnaturally large value as compared with the original image data, and a large edge is created at a boundary between the original image region and the extension region.
When resolution conversion is performed on the image consisting of the target block and the extension region by the IM-GPDCT method, large winding is created near the edge as shown in c). In other words, the IM-GPDCT method is intended to restore a high frequency component that is lost during a sampling operation, and therefore a high frequency component particularly exists at the edge portion and winding is created because of an attempt to restore the high frequency component.
Especially when blocks are cut out and then combined together after resolution conversion in order to make the processing faster, ringing is caused in each block. Therefore, a block noise is caused in a conspicuous manner in the conventional technique.
FIG. 20 shows charts for describing problems with a case where image data L of the extension region is set to 0 in the conventional technique.
In this case as well, the image data of L=0 is added as the extension region as shown in b), and therefore an excessive edge is created at a boundary between the extension region and the original image region and unnatural winding is found when the IM-GPDCT method is adopted.
FIG. 21 shows charts for describing problems with a case where the image data of the extension region is set to the average value of image data included in the target block in the conventional technique.
As shown in b), an edge portion is created between the extension region and the original image region even when the average value of image data in the original image region of the target block is used for the extension region. Further, the IM-GPDCT processing causes unnatural winding and an image noise as shown in c).
The IM-GPDCT method also has relatively long processing time. When the Bilinear method, for example, which has relatively short processing time, is used, however, the high frequency component of an image cannot be restored.
The present invention was made to solve the problems above and its first object is to prevent noise generation in a pixel interpolating device.
A second object of the present invention is to improve processing speed in a pixel interpolating device capable of restoring a high frequency component.
In order to achieve the objects above, according to one aspect of the present invention, a pixel interpolating device for interpolating a pixel by restoring a lost high frequency component under the two restrictive conditions that information on a passing frequency band is correct and an expanse of an image is limited in a process in which normal transformation and inverse transformation of an image are repeated by orthogonal transformation includes a cutting-out unit for cutting out a target block from an original image, and a setting unit for setting the data of an extension region of the target block, which is required for pixel interpolation, based on image data peripheral to the target block of the original image.
According to the present invention, noise generation can be prevented in the pixel interpolating device.
According to another aspect of the present invention, a pixel interpolating device for interpolating a pixel in an input image includes a determining unit for determining whether an edge portion exists in the input image, and a switching unit for switching a pixel interpolation method for the input image based on the determination result of the determining unit.
According to the present invention, the pixel interpolation method is switched based on the determination result as to whether an edge portion exists, and therefore the processing speed of the device can be improved.
According to still another aspect of the present invention, a pixel interpolation method includes the steps of a) cutting out a target block from original image data, b) adding an extension region around the target block to obtain an extension block, the data of the extension region being set based on image data peripheral to the target block of the original image, c) carrying out DCT transform on the image data of the extension region to obtain a frequency component, d) extending the obtained frequency component to a high frequency region and setting the initial value of the frequency component of the high frequency region to 0, e) carrying out inverse DCT transform on the frequency component obtained by extending the frequency component in d) to obtain the image data of a magnified extension block, the magnified extension block including a magnified target block at the center of the magnified extension block, f) setting data based on the image data peripheral to the target block of the original image for a peripheral region of the magnified target block in the magnified extension block, g) carrying out DCT transform on the image data of the extension block obtained in f) to obtain a frequency component, h) substituting a low frequency region of the frequency component obtained in g) by the frequency component obtained in c), and i) carrying out inverse DCT transform on the frequency component obtained in h) to obtain image data, magnified image data being obtained by repeating the steps from f) to i) a prescribed number of times.
According to still another aspect of the present invention, a pixel interpolating method includes the steps of determining whether an edge portion exists in an input image, selecting one of the first and second pixel interpolation methods based on the determination result, and carrying out pixel interpolation of the input image by the selected pixel interpolation method.
According to still another aspect of the present invention, an image processing apparatus for interpolating a pixel by restoring a high frequency component by repeating normal transformation and inverse transformation of an image through orthogonal transformation includes a cutting-out unit for cutting out a target block from original image data, a setting unit for setting an extension block by adding an extension region around the target block, the data of the extension region being determined based on image data peripheral to the target block of the original image, a transforming unit for carrying out DCT transform on the image data of the extension block to obtain a frequency component, a frequency extending unit for extending the obtained frequency component to a high frequency region and setting a prescribed value as the initial value of the frequency component of the high frequency region, and an inverse transforming unit for carrying out inverse DCT on the frequency component obtained by extending the frequency region to obtain magnified image data.
According to still another aspect of the present invention, a pixel interpolating device includes a first pixel interpolating unit for interpolating a pixel by restoring a high frequency component of an image, which is lost during sampling, a second pixel interpolating unit different from the first pixel interpolating unit, a determining unit for determining whether an edge portion exists in an input image, and a selecting unit for selecting one of the first and second pixel interpolating units based on the determination result.